Project 3 - Part 1-4

FFT and Aligning Images

Read in cell images and create other images from other image.

Read in and create overlapping crops and images from the drone image as another example.

1) FFTs

I followed the book for how to do the low pass filter in the Fourier domain and display multiple images power spectrums, the filters, and the resulting image back in the spatial domain after it has been filtered in the Fourier domain. I implemented the main 3 types of low pass filters described in the textbook which are the ideal low pass filter (ILPF), gaussian low pass filter (GLPF), and Butterworth low pass filter (BLPF). The ideal low pass filter has a very sharp transition like a circle while the gaussian and Butterworth filters have additional parameters that can make a softer transition as shown in the examples below.

Examples of the 3 low pass filters: Ideal, Gaussian, and Butterworth with different parameters so you can see what the filters look like. You can see the the Ideal LPF has a very sharp change while the Gaussian and Butterworth LPF can be more gradual depending on what parameters are passed to the filters.

Down below I display the original images, the power spectrum and the log of the power spectrum of the FFT of the images. I also display the filter and apply that filter in the Fourier domain. I do the padding because that is what is described in the course textbook in section (4.7 The Basics of Filtering in the Frequency Domain, see Figure 4.35 and above it where they describe the process of filtering in the Fourier Domain if needed). It sounds like it is not mandatory but it can be used to apply different effects to the image. You can see that I organize the frequencies so that zero is in the middle of the FFT. You don't see much only looking at the power spectrum, but you do see more information when you look at the log of the power spectrum which has some strong lines. I also show the Gaussian low pass filter that is used to filter in Fourier domain and the output image after the filtering. You can see that since it removes some of the higher frequencies that the image becomes slightly blurred and more soft.

Down below you can see the process as described above except on the other images. You can see that some of the log power spectrums for some of these images have additional vertical and horizontal lines for some of the stronger lines in the image. There is also a few other smaller lines in the log power spectrum that somewhat mirror other lines in the spatial domain. The Gaussian low pass filter has the same impact of removing some of the higher frequencies and blurring the image once we convert it back to the spatial domain.

The below chart shows the impact of the three different low pass filters in the Fourier domain with different parameters on different images. Its interesting to note the slight ringing effect the top ideal LPF has on the image because of the sharp edge in the filter. The lower Gaussian filter really blurs the image while the higher parameters for the Gaussian filter just slightly blurs the image. It's the same for the Butterworth filter, depending on what parameters you pass to the function.

2) Phase Correlation

I implemented the phase correlation with the low pass filter. I show examples of the two original images, the phase correlation output, and what it looks like with different low pass filters and parameters. You can see in the raw phase correlation that there is usually a pixel that is brighter than the rest that shows the offset between the images. You can see that this offset actually becomes easier to see with some of the Ideal, Gaussian, and Butterworth low pass filters. This is very obvious for the last three where the bright pixel becomes more obvious to see with the LPF. On the top two, you can see the brightest pixel on the top left and then on the second one the brightest pixel is on the bottom right.

Down below I display a comparison on changing the parameters to the filters when doing the phase correlation and the output. You can see that as you change the parameters to the filters is really does highlight where the that peak in the phase correlation is, so there is probably an optimal low pass params for certain types of images when doing the filtering with the phase correlation.

3) Peak Finding

I wrote out the functionality to perform the phase correlation and then find the pixel location with the maximal value. To then illustrate whether that implementation was correct I also align the images by resolving the four cases of ambiuity (as described in part 4 Mosaic Building). That way I can align the image using the peak found in the phase correlation and display the images together. You can see from the examples below that for my images using the pixel location with the maximal value works extremely well for aligning these images and it is very satisfying to see the images properly align. In my testing for my images, it seemed like using a simple threshold of 0.01 worked best for finding the peak value of the phase correlation and determining whether or not the images overlap. In practice, looking at the connected components from the phase correlation image thresholded at some value and then averaging the magnitude and position within those components in order to find the best peak to use for the two images works the best if you have correct thresholds. This is especially true depending on what low pass filter you use during the phase correlation and what parameters you pass to that filter. For example, if you look at the examples above you can see that certain filters make it so that the bright area in the phase correlation is a lot larger and consists of multiple pixels. In this scenario the base thing to do would be to use the thresholded connected components and do averaging on the position within those components find the best peak within that patch. Overall for most scenarios it seems like just using the pixel location with the maximal value performs very well as a simpler option to tune and use for these images.

Examples of peak finding by using the pixel location with the maximal value. I also printed and tested different thresholds for determining if the two images actually overlap. These examples below are specifically images that do overlap. I found that a value of 0.01 seemed to be a good threshold for highest pixel value for determining whether or not the images actually overlap. You can see the max values below which are between 0.02 - 0.26.

NOTE: Examples of Images that do not overlap below.

Examples of images that do not overlap for finding a good threshold of maximum value to be below a certain threshold. You can see that for these three examples that the max value is only up to around 0.007 which is under the threshold of 0.01. NOTE: I still display the images aligned improperly just to demonstrate that they do not overlap properly, but I print out that the images do not overlap based on that max val being below the threshold.

Examples of peak finding using connected components above a threshold of 0.01 and then averaging the magnitude and position within the found components in order to find the best peak for the offset between the two images.

4) Mosaic Building

I created a function called mosaic_image_from_file() which will read in a list of images from a file, and then will create a mosaic from them and output them to a file. I also display the mosaic'd image in the notebook. The mosaic_image_from_file uses the mosaic_images function which computes the matrix of phase correlation and cross correlation from the aligned images and then uses that in order to align and put all images onto the canvas. You can see the mosaic created from the cell images and the beach trees images down below. In order to align the images properly, I use the technique in the last paragraph of (4 - Mosaic Building) on the assignment and resolve the four cases of ambiguity after the peak in phase correlation is found. I then computer the correlation on the overlapping patches in the spatial domain as described in the image on the homework. This helps resolve the four cases of ambiguity and figure out how to align the images.

Below is an example of building the mosaic from images already read in an array in memory. You can see that it computes the cross correlation matrix and then uses that to select the anchor image and build the mosaic from there. I resolve the four cases of ambiguity as described above and in the assignment information.

Below is the example of building a function that reads in a set of images specified in a file and builds and then saves a mosaic out to the desired location, based on the parameters that were passed into the function.

Below is the example of building a function that reads in a set of images specified in a file for the drone image and builds and then saves a mosaic out to the desired location, based on the parameters that were passed into the function.